
<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta charset="utf-8" />
    <title>32.5. token — Constants used with Python parse trees &#8212; Python 3.6.10 documentation</title>
    <link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    
    <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/underscore.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <script type="text/javascript" src="../_static/language_data.js"></script>
    
    <script type="text/javascript" src="../_static/sidebar.js"></script>
    
    <link rel="search" type="application/opensearchdescription+xml"
          title="Search within Python 3.6.10 documentation"
          href="../_static/opensearch.xml"/>
    <link rel="author" title="About these documents" href="../about.html" />
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="copyright" title="Copyright" href="../copyright.html" />
    <link rel="next" title="32.6. keyword — Testing for Python keywords" href="keyword.html" />
    <link rel="prev" title="32.4. symbol — Constants used with Python parse trees" href="symbol.html" />
    <link rel="shortcut icon" type="image/png" href="../_static/py.png" />
    <link rel="canonical" href="https://docs.python.org/3/library/token.html" />
    
    <script type="text/javascript" src="../_static/copybutton.js"></script>
    <script type="text/javascript" src="../_static/switchers.js"></script>
    
    
 

  </head><body>
  
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="../py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="keyword.html" title="32.6. keyword — Testing for Python keywords"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="symbol.html" title="32.4. symbol — Constants used with Python parse trees"
             accesskey="P">previous</a> |</li>
        <li><img src="../_static/py.png" alt=""
                 style="vertical-align: middle; margin-top: -1px"/></li>
        <li><a href="https://www.python.org/">Python</a> &#187;</li>
        <li>
          <span class="language_switcher_placeholder">en</span>
          <span class="version_switcher_placeholder">3.6.10</span>
          <a href="../index.html">Documentation </a> &#187;
        </li>

          <li class="nav-item nav-item-1"><a href="index.html" >The Python Standard Library</a> &#187;</li>
          <li class="nav-item nav-item-2"><a href="language.html" accesskey="U"><span class="section-number">32. </span>Python Language Services</a> &#187;</li>
    <li class="right">
        

    <div class="inline-search" style="display: none" role="search">
        <form class="inline-search" action="../search.html" method="get">
          <input placeholder="Quick search" type="text" name="q" />
          <input type="submit" value="Go" />
          <input type="hidden" name="check_keywords" value="yes" />
          <input type="hidden" name="area" value="default" />
        </form>
    </div>
    <script type="text/javascript">$('.inline-search').show(0);</script>
         |
    </li>

      </ul>
    </div>    

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="module-token">
<span id="token-constants-used-with-python-parse-trees"></span><h1><span class="section-number">32.5. </span><a class="reference internal" href="#module-token" title="token: Constants representing terminal nodes of the parse tree."><code class="xref py py-mod docutils literal notranslate"><span class="pre">token</span></code></a> — Constants used with Python parse trees<a class="headerlink" href="#module-token" title="Permalink to this headline">¶</a></h1>
<p><strong>Source code:</strong> <a class="reference external" href="https://github.com/python/cpython/tree/3.6/Lib/token.py">Lib/token.py</a></p>
<hr class="docutils" />
<p>This module provides constants which represent the numeric values of leaf nodes
of the parse tree (terminal tokens).  Refer to the file <code class="file docutils literal notranslate"><span class="pre">Grammar/Grammar</span></code>
in the Python distribution for the definitions of the names in the context of
the language grammar.  The specific numeric values which the names map to may
change between Python versions.</p>
<p>The module also provides a mapping from numeric codes to names and some
functions.  The functions mirror definitions in the Python C header files.</p>
<dl class="data">
<dt id="token.tok_name">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">tok_name</code><a class="headerlink" href="#token.tok_name" title="Permalink to this definition">¶</a></dt>
<dd><p>Dictionary mapping the numeric values of the constants defined in this module
back to name strings, allowing more human-readable representation of parse trees
to be generated.</p>
</dd></dl>

<dl class="function">
<dt id="token.ISTERMINAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ISTERMINAL</code><span class="sig-paren">(</span><em class="sig-param">x</em><span class="sig-paren">)</span><a class="headerlink" href="#token.ISTERMINAL" title="Permalink to this definition">¶</a></dt>
<dd><p>Return true for terminal token values.</p>
</dd></dl>

<dl class="function">
<dt id="token.ISNONTERMINAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ISNONTERMINAL</code><span class="sig-paren">(</span><em class="sig-param">x</em><span class="sig-paren">)</span><a class="headerlink" href="#token.ISNONTERMINAL" title="Permalink to this definition">¶</a></dt>
<dd><p>Return true for non-terminal token values.</p>
</dd></dl>

<dl class="function">
<dt id="token.ISEOF">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ISEOF</code><span class="sig-paren">(</span><em class="sig-param">x</em><span class="sig-paren">)</span><a class="headerlink" href="#token.ISEOF" title="Permalink to this definition">¶</a></dt>
<dd><p>Return true if <em>x</em> is the marker indicating the end of input.</p>
</dd></dl>

<p>The token constants are:</p>
<dl class="data">
<dt id="token.ENDMARKER">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ENDMARKER</code><a class="headerlink" href="#token.ENDMARKER" title="Permalink to this definition">¶</a></dt>
<dt id="token.NAME">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">NAME</code><a class="headerlink" href="#token.NAME" title="Permalink to this definition">¶</a></dt>
<dt id="token.NUMBER">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">NUMBER</code><a class="headerlink" href="#token.NUMBER" title="Permalink to this definition">¶</a></dt>
<dt id="token.STRING">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">STRING</code><a class="headerlink" href="#token.STRING" title="Permalink to this definition">¶</a></dt>
<dt id="token.NEWLINE">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">NEWLINE</code><a class="headerlink" href="#token.NEWLINE" title="Permalink to this definition">¶</a></dt>
<dt id="token.INDENT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">INDENT</code><a class="headerlink" href="#token.INDENT" title="Permalink to this definition">¶</a></dt>
<dt id="token.DEDENT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">DEDENT</code><a class="headerlink" href="#token.DEDENT" title="Permalink to this definition">¶</a></dt>
<dt id="token.LPAR">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LPAR</code><a class="headerlink" href="#token.LPAR" title="Permalink to this definition">¶</a></dt>
<dt id="token.RPAR">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">RPAR</code><a class="headerlink" href="#token.RPAR" title="Permalink to this definition">¶</a></dt>
<dt id="token.LSQB">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LSQB</code><a class="headerlink" href="#token.LSQB" title="Permalink to this definition">¶</a></dt>
<dt id="token.RSQB">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">RSQB</code><a class="headerlink" href="#token.RSQB" title="Permalink to this definition">¶</a></dt>
<dt id="token.COLON">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">COLON</code><a class="headerlink" href="#token.COLON" title="Permalink to this definition">¶</a></dt>
<dt id="token.COMMA">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">COMMA</code><a class="headerlink" href="#token.COMMA" title="Permalink to this definition">¶</a></dt>
<dt id="token.SEMI">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">SEMI</code><a class="headerlink" href="#token.SEMI" title="Permalink to this definition">¶</a></dt>
<dt id="token.PLUS">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">PLUS</code><a class="headerlink" href="#token.PLUS" title="Permalink to this definition">¶</a></dt>
<dt id="token.MINUS">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">MINUS</code><a class="headerlink" href="#token.MINUS" title="Permalink to this definition">¶</a></dt>
<dt id="token.STAR">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">STAR</code><a class="headerlink" href="#token.STAR" title="Permalink to this definition">¶</a></dt>
<dt id="token.SLASH">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">SLASH</code><a class="headerlink" href="#token.SLASH" title="Permalink to this definition">¶</a></dt>
<dt id="token.VBAR">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">VBAR</code><a class="headerlink" href="#token.VBAR" title="Permalink to this definition">¶</a></dt>
<dt id="token.AMPER">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">AMPER</code><a class="headerlink" href="#token.AMPER" title="Permalink to this definition">¶</a></dt>
<dt id="token.LESS">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LESS</code><a class="headerlink" href="#token.LESS" title="Permalink to this definition">¶</a></dt>
<dt id="token.GREATER">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">GREATER</code><a class="headerlink" href="#token.GREATER" title="Permalink to this definition">¶</a></dt>
<dt id="token.EQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">EQUAL</code><a class="headerlink" href="#token.EQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.DOT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">DOT</code><a class="headerlink" href="#token.DOT" title="Permalink to this definition">¶</a></dt>
<dt id="token.PERCENT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">PERCENT</code><a class="headerlink" href="#token.PERCENT" title="Permalink to this definition">¶</a></dt>
<dt id="token.LBRACE">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LBRACE</code><a class="headerlink" href="#token.LBRACE" title="Permalink to this definition">¶</a></dt>
<dt id="token.RBRACE">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">RBRACE</code><a class="headerlink" href="#token.RBRACE" title="Permalink to this definition">¶</a></dt>
<dt id="token.EQEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">EQEQUAL</code><a class="headerlink" href="#token.EQEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.NOTEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">NOTEQUAL</code><a class="headerlink" href="#token.NOTEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.LESSEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LESSEQUAL</code><a class="headerlink" href="#token.LESSEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.GREATEREQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">GREATEREQUAL</code><a class="headerlink" href="#token.GREATEREQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.TILDE">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">TILDE</code><a class="headerlink" href="#token.TILDE" title="Permalink to this definition">¶</a></dt>
<dt id="token.CIRCUMFLEX">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">CIRCUMFLEX</code><a class="headerlink" href="#token.CIRCUMFLEX" title="Permalink to this definition">¶</a></dt>
<dt id="token.LEFTSHIFT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LEFTSHIFT</code><a class="headerlink" href="#token.LEFTSHIFT" title="Permalink to this definition">¶</a></dt>
<dt id="token.RIGHTSHIFT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">RIGHTSHIFT</code><a class="headerlink" href="#token.RIGHTSHIFT" title="Permalink to this definition">¶</a></dt>
<dt id="token.DOUBLESTAR">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">DOUBLESTAR</code><a class="headerlink" href="#token.DOUBLESTAR" title="Permalink to this definition">¶</a></dt>
<dt id="token.PLUSEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">PLUSEQUAL</code><a class="headerlink" href="#token.PLUSEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.MINEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">MINEQUAL</code><a class="headerlink" href="#token.MINEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.STAREQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">STAREQUAL</code><a class="headerlink" href="#token.STAREQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.SLASHEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">SLASHEQUAL</code><a class="headerlink" href="#token.SLASHEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.PERCENTEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">PERCENTEQUAL</code><a class="headerlink" href="#token.PERCENTEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.AMPEREQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">AMPEREQUAL</code><a class="headerlink" href="#token.AMPEREQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.VBAREQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">VBAREQUAL</code><a class="headerlink" href="#token.VBAREQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.CIRCUMFLEXEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">CIRCUMFLEXEQUAL</code><a class="headerlink" href="#token.CIRCUMFLEXEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.LEFTSHIFTEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">LEFTSHIFTEQUAL</code><a class="headerlink" href="#token.LEFTSHIFTEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.RIGHTSHIFTEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">RIGHTSHIFTEQUAL</code><a class="headerlink" href="#token.RIGHTSHIFTEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.DOUBLESTAREQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">DOUBLESTAREQUAL</code><a class="headerlink" href="#token.DOUBLESTAREQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.DOUBLESLASH">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">DOUBLESLASH</code><a class="headerlink" href="#token.DOUBLESLASH" title="Permalink to this definition">¶</a></dt>
<dt id="token.DOUBLESLASHEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">DOUBLESLASHEQUAL</code><a class="headerlink" href="#token.DOUBLESLASHEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.AT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">AT</code><a class="headerlink" href="#token.AT" title="Permalink to this definition">¶</a></dt>
<dt id="token.ATEQUAL">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ATEQUAL</code><a class="headerlink" href="#token.ATEQUAL" title="Permalink to this definition">¶</a></dt>
<dt id="token.RARROW">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">RARROW</code><a class="headerlink" href="#token.RARROW" title="Permalink to this definition">¶</a></dt>
<dt id="token.ELLIPSIS">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ELLIPSIS</code><a class="headerlink" href="#token.ELLIPSIS" title="Permalink to this definition">¶</a></dt>
<dt id="token.OP">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">OP</code><a class="headerlink" href="#token.OP" title="Permalink to this definition">¶</a></dt>
<dt id="token.AWAIT">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">AWAIT</code><a class="headerlink" href="#token.AWAIT" title="Permalink to this definition">¶</a></dt>
<dt id="token.ASYNC">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ASYNC</code><a class="headerlink" href="#token.ASYNC" title="Permalink to this definition">¶</a></dt>
<dt id="token.ERRORTOKEN">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">ERRORTOKEN</code><a class="headerlink" href="#token.ERRORTOKEN" title="Permalink to this definition">¶</a></dt>
<dt id="token.N_TOKENS">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">N_TOKENS</code><a class="headerlink" href="#token.N_TOKENS" title="Permalink to this definition">¶</a></dt>
<dt id="token.NT_OFFSET">
<code class="sig-prename descclassname">token.</code><code class="sig-name descname">NT_OFFSET</code><a class="headerlink" href="#token.NT_OFFSET" title="Permalink to this definition">¶</a></dt>
<dd><div class="versionchanged">
<p><span class="versionmodified changed">Changed in version 3.5: </span>Added <a class="reference internal" href="#token.AWAIT" title="token.AWAIT"><code class="xref py py-data docutils literal notranslate"><span class="pre">AWAIT</span></code></a> and <a class="reference internal" href="#token.ASYNC" title="token.ASYNC"><code class="xref py py-data docutils literal notranslate"><span class="pre">ASYNC</span></code></a> tokens. Starting with
Python 3.7, “async” and “await” will be tokenized as <a class="reference internal" href="#token.NAME" title="token.NAME"><code class="xref py py-data docutils literal notranslate"><span class="pre">NAME</span></code></a>
tokens, and <a class="reference internal" href="#token.AWAIT" title="token.AWAIT"><code class="xref py py-data docutils literal notranslate"><span class="pre">AWAIT</span></code></a> and <a class="reference internal" href="#token.ASYNC" title="token.ASYNC"><code class="xref py py-data docutils literal notranslate"><span class="pre">ASYNC</span></code></a> will be removed.</p>
</div>
</dd></dl>

</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h4>Previous topic</h4>
  <p class="topless"><a href="symbol.html"
                        title="previous chapter"><span class="section-number">32.4. </span><code class="xref py py-mod docutils literal notranslate"><span class="pre">symbol</span></code> — Constants used with Python parse trees</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="keyword.html"
                        title="next chapter"><span class="section-number">32.6. </span><code class="xref py py-mod docutils literal notranslate"><span class="pre">keyword</span></code> — Testing for Python keywords</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="../bugs.html">Report a Bug</a></li>
      <li>
        <a href="https://github.com/python/cpython/blob/3.6/Doc/library/token.rst"
            rel="nofollow">Show Source
        </a>
      </li>
    </ul>
  </div>
        </div>
      </div>
      <div class="clearer"></div>
    </div>  
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="../py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="keyword.html" title="32.6. keyword — Testing for Python keywords"
             >next</a> |</li>
        <li class="right" >
          <a href="symbol.html" title="32.4. symbol — Constants used with Python parse trees"
             >previous</a> |</li>
        <li><img src="../_static/py.png" alt=""
                 style="vertical-align: middle; margin-top: -1px"/></li>
        <li><a href="https://www.python.org/">Python</a> &#187;</li>
        <li>
          <span class="language_switcher_placeholder">en</span>
          <span class="version_switcher_placeholder">3.6.10</span>
          <a href="../index.html">Documentation </a> &#187;
        </li>

          <li class="nav-item nav-item-1"><a href="index.html" >The Python Standard Library</a> &#187;</li>
          <li class="nav-item nav-item-2"><a href="language.html" ><span class="section-number">32. </span>Python Language Services</a> &#187;</li>
    <li class="right">
        

    <div class="inline-search" style="display: none" role="search">
        <form class="inline-search" action="../search.html" method="get">
          <input placeholder="Quick search" type="text" name="q" />
          <input type="submit" value="Go" />
          <input type="hidden" name="check_keywords" value="yes" />
          <input type="hidden" name="area" value="default" />
        </form>
    </div>
    <script type="text/javascript">$('.inline-search').show(0);</script>
         |
    </li>

      </ul>
    </div>  
    <div class="footer">
    &copy; <a href="../copyright.html">Copyright</a> 2001-2020, Python Software Foundation.
    <br />
    The Python Software Foundation is a non-profit corporation.
    <a href="https://www.python.org/psf/donations/">Please donate.</a>
    <br />
    Last updated on Apr 30, 2020.
    <a href="../bugs.html">Found a bug</a>?
    <br />
    Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 2.3.1.
    </div>

  </body>
</html>